Image processing apparatus and method, and program and recording medium

ABSTRACT

In a camera which electronically realizes panning, tilting, and zooming, on the basis of the luminance detected in each frame period (n−1), a condition of exposure for two frame periods thereafter (n+1), and digital gain for three frame periods thereafter (n+2) are set. When an instruction for changing the extracted region is received in a certain frame period (n−1), the region extracted from the image is changed three frame periods thereafter. It is possible to achieve a stable exposure control by which the luminance of the image varies little even when the region extracted from the image is switched.

TECHNICAL FIELD

The present invention relates to an image processing apparatus andmethod, and in particular to an image processing apparatus and methodwhich is suitable for application to a surveillance camera, and in whicha region which is a part of an image obtained by image capture with awide angle of view using a wide-angle lens, e.g., a fisheye lens, isextracted, and enlarged, and displayed, and at the same time theposition and the size of the extracted region are switched to performpanning, tilting or zooming of the image, and in particular to itsexposure control technique. The present invention also relates to aprogram for having a computer to implement part or the entirety of theelements of the image processing apparatus, or part or the entirety ofthe processes in the image processing method, and a recording medium inwhich the program is stored.

BACKGROUND ART

A camera by which a part of the image obtained by image capture isextracted and displayed, and the position and the size of the extractedregion can be changed is disclosed in patent reference 1. In patentreference 1, when a part of the image is extracted and displayed, theexposure control is performed using the luminance information of theentire screen, rather than the extracted region, so as to reduce theeffects of the variation in the luminance of a localized object.

PRIOR ART REFERENCES Patent References

Patent reference 1: Japanese Patent Application Publication No.H04-23576 (pages 14-20, FIG. 1).

SUMMARY OF THE INVENTION Problem to be Solved by the Invention

When the exposure control is performed by constantly using the imagebefore the extraction, as described in patent reference 1, the luminanceof the image obtained as a result of the exposure control is stable,being free from substantial variation; but, with an object condition inwhich the luminance varies from one region to another, if only alow-luminance region is extracted and displayed, the screen as a wholebecomes dark, and the visibility of the image is lowered; while if ahigh luminance region only is extracted and displayed, blown-outhighlights in the image, or saturation of the pixel values may occur,and the visibility of the objects is lowered.

To solve this problem, it is necessary to perform the exposureprocessing by referring to the extracted image region; but when thesetting of the panning, tilting, or zooming of the image is switched ata high rate, the image region referred to is changed frequently, so thatthe luminance information varies upon all such occasions, with theresult that the exposure control process using the luminance informationbecomes unstable, and the brightness of the displayed image is changedat the time of switching.

In view of the above, an object of the present invention is to providean image processing apparatus in which extraction, enlargement, anddistortion correction are performed while changing the position and thesize of the extracted region of the image obtained by image captureusing a lens with a wide angle of view, thereby to electronicallyperform panning, tilting, or zooming, and which can perform stableexposure control with little variation in the luminance of the image,even when the region extracted from the image is switched.

Means for Solving the Problem

An image processing apparatus according to the present inventioncomprises:

a digital gain application unit for multiplying, by a digital gain, acaptured image signal output from an image capture unit which capturesimages in units of frame periods, to generate a luminance-adjustedcaptured image signal;

a luminance detection unit for detecting luminance of each of aplurality of regions which respectively form parts of a captured imagerepresented by the captured image signal generated by said digital gainapplication unit;

an extraction unit for selecting one of the plurality of regions of thecaptured image, in accordance with designation information designating aregion to be extracted, extracting an image of the selected region, andperforming distortion correction; and

a control unit for, on a basis of the luminance detected by saidluminance detection unit, setting a condition of exposure in said imagecapture unit, and setting the digital gain used in said digital gainapplication unit; wherein

when the designation information is changed from information designatinga first region to information designating a second region, in a firstframe period,

said control unit

changes the luminance used for setting said condition of the exposureand said digital gain in a frame period immediately following said firstframe period, from the luminance of said first region to the luminanceof said second region, and

instructs said extraction unit to change the region extracted from thecaptured image signal, from said first region to said second region,upon expiration of three frame periods after said first frame period.

Effects of the Invention

According to the present invention, it is possible to achieve exposurecontrol which is stable, and which suffers little variation in theluminance of the image even when the extracted region of the image isswitched.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an image generating apparatus providedwith an image processing apparatus according to a first embodiment ofthe present invention.

FIG. 2 is a diagram showing an example of projection of an image inimage capture using a wide-angle lens, and distortion correction.

FIGS. 3(a) to (d) are diagrams showing projection of an image from animaginary sphere onto an image plane, division into a plurality ofregions, and combination of images of the plurality of regions.

FIGS. 4(a) to (k) are diagrams showing an example of a sequence ofprocesses in various parts in the first embodiment.

FIGS. 5(a) to (k) are diagrams showing a sequence of processes invarious parts in a conventional image processing apparatus.

FIGS. 6(a) to (k) are diagrams showing another example of a sequence ofprocesses in various parts in the first embodiment.

FIGS. 7(a) to (d) are diagrams showing the relations between anextracted region and a plurality of sections used for the calculation ofthe luminance of the extracted region in a second embodiment of thepresent invention.

MODE FOR CARRYING OUT THE INVENTION First Embodiment

FIG. 1 shows an image generating apparatus 10 provided with an imageprocessing apparatus according to a first embodiment of the presentinvention. The illustrated image generating apparatus 10 is providedwith a wide-angle image acquisition unit 12 and an image processingapparatus 14.

The wide-angle image acquisition unit 12 includes a wide-angle lens 20and an image capture unit 30.

The image processing apparatus 14 includes a digital gain applicationunit 40, a luminance detection unit 50, an image generating unit 60, anextraction unit 70, and an image combining unit 80.

The wide-angle lens 20 is constituted, for example, of a fisheye lens,and forms an image of a wide angle of view on an image plane of theimage capture unit 30.

Image capture by use of a fisheye lens can be represented as projectionof an object OJ in the real space onto an imaginary sphere PL, andprojection from the imaginary sphere PL onto a planar image plane SL, asshown in FIG. 2. The image on the image plane SL undergoes correctiondepending on the intended use of the image, for instance, correction(distortion correction) for altering the image to be closer to an imageobtained by ordinary image capture, that is, an image NL obtained byperspective projection.

The image capture unit 30 is constituted of an image sensor forconverting the image formed, into electronic data, and outputting theelectronic data. The image capture unit 30 generates a captured image inunits of predetermined frame periods, that is, every frame period, and acaptured image signal representing the captured image generated bycharge accumulation due to exposure in each frame period is output in anext frame period. The image capture unit 30 has a Bayer arrangement ofR pixels, G pixels, and B pixels, and the captured image signal iscomposed of pixel values (color signals) respectively obtained at the Rpixels, G pixels and B pixels. The exposure time and the gain (analoggain) of the image capture unit 30 are variable, and by varying theexposure time and the gain, the brightness of the captured image can beadjusted.

The digital gain application unit (digital amplifier) 40 multiplies thecaptured image signal from the image capture unit 30 by a gain (digitalgain), that is, it applies a gain factor. The gain used for themultiplication is variable, and, as in the image capture unit 30, bycontrolling the gain, the brightness of the image output from thedigital gain application unit 40 can be adjusted. As was describedabove, the captured image signal generated by carrying out the exposurewith an exposure condition in a certain frame period (first frameperiod) in the image capture unit 30 is output in the next frame period(second frame period). The multiplication by the digital gain at thedigital gain application unit 40 in each frame period, is made on thecaptured image signal that is generated as a result of the exposure inthe preceding frame period in the image capture unit 30, and input fromthe image capture unit 30.

The control unit 100 divides the image on the image plane SL into aplurality of regions, and instructs the luminance detection unit 50 toperform luminance detection of each of the regions formed by thedivision.

The luminance detection unit 50 performs the detection of luminance ofeach of the regions formed by the division by the control unit 100. Thedetection of luminance is implemented by calculation of an average pixelvalue for each region. In the present embodiment, the luminance isdetected for all of the divided regions in each frame period.

The calculation of the average pixel value for each region in each frameperiod is completed before the end of the frame period in question, thatis, before the next frame period begins.

The image generating unit 60 generates, by color interpolation process,signals of color components which are missing at each pixel position inthe image output from the digital gain application unit 40, because ofthe Bayer arrangement of the pixels of the image capture unit 30, tothereby generate a video signal having signals of all the colorcomponents at each pixel position, and also performs image processingsuch as, color correction, and edge enhancement, in consideration of theilluminance conditions of the objects.

The extraction unit 70 receives the image output from the imagegenerating unit 60, selects one of the regions among the plurality ofregions formed by the division by the control unit 100, extracts animage of the selected region, and performs distortion correction andenlargement.

The image combining unit 80 combines the images of a plurality ofregions successively output from the extraction unit 70, to form asingle image corresponding to an angle of view wider than the angle ofview corresponding to each of the regions.

The division of the image by the control unit 100 is, for example, asindicated by dotted lines in FIG. 2 and FIG. 3(b), if shown inconnection with the image on the image plane SL. The extraction by theextraction unit 70 is a process of extracting one of the regions, e.g.,the regions R1, R2, . . . , shown in FIG. 3(b).

The regions R1, R2, . . . on the image plane SL in FIG. 3(b) correspondto the regions R1, R2, . . . on the imaginary sphere PL.

The distortion correction by the extraction unit 70 is a process oftransforming the image on the image plane SL, to an image suitable forthe intended use, e.g., an image closer to an image NL (FIG. 2) obtainedby perspective projection.

The combination by the image combining unit 80 is a process ofconnecting the images of the plurality of regions, for example theimages of the regions R1 to R6 shown in FIG. 3(c), obtained by thesuccessive extraction and distortion correction by the extraction unit70, to form a single image as shown in FIG. 3(d). The regions which areconnected correspond to the regions in a range shown by a contour CR inFIG. 3(a).

The control unit 100 receives designation information DR designating anextracted region, from outside, performs control, in accordance with thedesignation, over the image capture unit 30, the digital gainapplication unit 40, the luminance detection unit 50, the imagegenerating unit 60, the extraction unit 70, and the image combining unit80, with respect to the switching of the extracted region, and thecombination of the images.

The designation of the extracted region by use of the designationinformation DR includes information indicating the position of theextracted region, e.g., the position of its center, and informationindicating the size of the extracted region. When identification numbersare pre-assigned to the plurality of regions, information indicating theidentification number is also included, and based on this information,the control unit 100 identifies the region to be extracted.

The control unit 100 performs control over the image generating unit 60with regard to image processing, such as color correction, and edgeenhancement, by setting conditions for these image processing inconsideration of the illuminance conditions of the objects.

The illuminance conditions of the objects may, for example, be thoseestimated from the luminance value detected by the luminance detectionunit 50.

The control unit 100 also performs control over the image combining unit80 with regard to the combination of the images.

The control unit 100 also acquires the luminance detection result (pixelaverage value) output from the luminance detection unit 50, and,performs, based on the thus-acquired luminance detection result, thesetting of the exposure condition in the image capture unit 30, andsetting of the digital gain in the digital gain application unit 40.

The setting of the exposure condition in the image capture unit 30includes setting of the exposure time and setting of the gain (analoggain).

FIGS. 4(a) to (k) show a sequence of operations of the control unit 100,the image capture unit 30, the digital gain application unit 40, and theluminance detection unit 50.

FIGS. 4(a) to (k) show operations during eight frame periods, from framen−4 to frame n+3, at the time of switching from a certain region (firstregion) A, to another region (second region) B. Each of the regions Aand B is, for example, one of the regions R1 to R6.

FIGS. 4(a) to (k) assume a situation in which an instruction forswitching the region is given in frame n−1.

FIG. 4(a) shows regions extracted by the extraction unit 70. In theillustrated example, the region A is extracted up to frame n+1, and theregion B is extracted in and after frame n+2.

FIGS. 4(b), (c), (d) and (e) show processes by the control unit 100.

FIG. 4(b) shows the process of acquiring the result of the luminancedetection from the luminance detection unit 50.

In the present embodiment, the luminance detection is made for all theregions in each frame, as was described above, but the control unit 100acquires the detection result of just one of the regions, and uses itfor the computation of the set values.

In the illustrated example, the luminance detection result of the regionA is acquired up to frame n−1, and the luminance detection result of theregion B is acquired in and after frame n.

FIG. 4(c) shows the computation of the set values based on the result ofthe luminance detection acquired by the process in FIG. 4(b); FIG. 4(d)shows the setting in the image capture unit 30, of the set values of theexposure condition (exposure time, and analog gain) determined as aresult of the computation in FIG. 4(c); and FIG. 4(e) shows the settingin the digital gain application unit 40, of the digital gain set valuedetermined as a result of the computation in FIG. 4(c).

FIG. 4(f) shows the valid image period VA and the vertical blankingperiod BL in each frame.

In FIGS. 4(b) to (e), the position, in the time axis direction, of arectangle representing each process schematically indicates the time atwhich each process is carried out.

In FIGS. 4(b) to (j), an arrow-headed line drawn from one process toanother process indicates transmission of information or a signalrepresenting the result obtained at each process, to another process.

The process of acquiring the result of the luminance detection in FIG.4(b) is carried out at the beginning of each frame. The luminancedetection result acquired in each frame is obtained by calculation of anaverage value of pixel values in the immediately preceding frame. Theregion of which the luminance detection result is to be acquired isdecided at the beginning of each frame, and, therefore, the luminancedetection is made for all the regions in the preceding frame.

The process of the set value computation in FIG. 4(c) is carried outafter the process of FIG. 4(a), within the same frame period.

The process of setting the exposure condition in the image capture unit30, and the process of setting the digital gain in the digital gainapplication unit shown in FIG. 4(d) and FIG. 4(e) are carried out afterthe process of FIG. 4(c), within the same frame period, specifically,during the vertical blanking period BL at the end of the frame period inquestion.

By carrying out the process of the setting during the blanking periodBL, it is possible to avoid changes in the brightness of the imagewithin the same frame, that might occur if the setting is changed duringthe valid image period VA.

FIG. 4(g) and FIG. 4(h) show processes in the image capture unit 30.

FIG. 4(g) shows exposure processes, while FIG. 4(h) shows data output.

FIG. 4(i) shows the data output from the digital gain application unit40, and FIG. 4(j) shows the detection of the luminance (measurement ofthe pixel values for that purpose) by the luminance detection unit 50.

In the present embodiment, as was described above, the luminancedetection is made for all the regions in each frame; this is indicatedby the characters “ALL REGIONS” in FIG. 4(j), and the region of whichthe luminance value detection result is acquired at the beginning of thenext frame is indicated in the parentheses after the characters “ALLREGIONS” in FIG. 4(j).

In the illustrated example, the luminance detection result of the regionA in each of the frames up to frame n−2 is acquired in the next frame,and the luminance detection result of the region B in each of the framesin and after frame n−1 is acquired at the beginning of the next frame.

What follows is a description on the processes in which the result ofthe luminance detection by the luminance detection unit 50 is used forthe setting of the exposure condition in the image capture unit 30, thesetting of the digital gain in the digital gain application unit 40, andthe data obtained with these settings is output (the luminance detectionresult is reflected in the output data) is explained.

For example, in frame n−4, the luminance detection is made for all theregions including the region A (FIG. 4(j)); the luminance detectionresult of the region A is acquired by the control unit 100 at thebeginning of frame n−3 (FIG. 4(b)); in the same frame n−3, thecomputation of the set values (FIG. 4(c)), the setting in the imagecapture unit 30 (FIGS. 4(d)), and the setting in the digital gainapplication unit 40 (FIG. 4(e)) are carried out; in response to thesesettings, the exposure (FIG. 4(g)) in the next frame n−2 is performed bythe image capture unit 30; the captured image signal generated as aresult of the exposure is output from the image capture unit 30 in framen−1 immediately following the above-mentioned next frame; and in thesame frame n−1, the multiplication by the digital gain at the digitalgain application unit 40 is performed (FIG. 4(i)).

Thus, the data obtained by the exposure in frame n−2 performed with theexposure condition set in frame n−3 is output from the image captureunit in frame n−1 (FIG. 4(h)), and the data is multiplied by the digitalgain which is set also in frame n−3.

There is a delay of two frame periods from the acquisition (frame n−3)of the luminance detection result to the output (frame n−1) of the datagenerated as a result of the exposure with the exposure condition setbased on the result of the acquisition, and the multiplication by thedigital gain set based on the result of the acquisition.

According to the present invention, if an instruction for switching tothe extracted region B (change of the designated region) is given in acertain frame, e.g., frame n−1 (FIG. 4(k)), the control unit 100switches the region of which the luminance detection result is acquired,to the region B, in the next frame n (acquires the luminance detectionresult of the region B in frame n−1); and, starting with frame n+2,which is two frames thereafter (three frames after the instruction forthe switching), the extracted region is switched to the region B (FIG.4(a)).

The reason why it is possible to switch the region of which theluminance detection result is acquired at the beginning of frame n,which immediately follows the frame in which the extracted regionswitching is instructed is that the luminance information detection ismade for all the regions in each frame.

In frame n+1 before the extracted region switching, the region A isextracted; in this frame n+1, the data obtained as a result of theexposure (frame n) with the exposure condition set based on theluminance detection result of the region A acquired in frame n−1, andthe multiplication (frame n+1) by the digital gain set based on theluminance detection result of the region A acquired in frame n−1, isoutput.

In frame n+2 after the extracted region switching, the region B isextracted; in this frame n+2, the data obtained as a result of theexposure (frame n+1) with the exposure condition set based on theluminance detection result of the region B acquired in frame n, and themultiplication (frame n+2) by the digital gain set based on theluminance detection result of the region B acquired in frame n.

Thus, before and after the extracted region switching, the data of eachextracted region is obtained as a result of the exposure and the digitalgain multiplication with the conditions set based on the luminancedetection result of the same region.

For the purpose of comparison, the processes in the conventionalconfiguration will be explained with reference to FIGS. 5(a) to (k).

In the conventional method, the luminance detection in each frame ismade only for the region being selected. For instance, it is assumedthat a switching demand is given in frame n−1 as shown in FIG. 5(k).

In this case, as shown in FIG. 5(j), the luminance of the region A isdetected up to frame n−1, and the luminance of the region B is detectedin and after frame n.

As a result, the luminance detection result of the region A is acquiredup to frame n, and the luminance detection result of the region B isacquired in and after frame n+1 (FIG. 5(b)).

Accordingly, up to frame n, the set value computation, the setting ofthe exposure condition, and the setting of the digital gain are madebased on the luminance detection result of the region A; in an afterframe n+1, the set value computation, the setting of the exposurecondition, and the setting of the digital gain are made based on theluminance detection result of the region B.

Furthermore, up to frame n+1, the exposure with the exposure conditionset based on the luminance detection result of the region A and themultiplication by the digital gain set based on the luminance detectionresult of the region A are performed; and, in and after frame n+2, theexposure with the exposure condition set based on the luminancedetection result of the region B and the multiplication by the digitalgain set based on the luminance detection result of the region B areperformed.

Because of these processes, the data (FIG. 5(h)) output from the imagecapture unit as a result of the exposure (FIG. 5(g)) with the exposurecondition set based on the luminance detection result of the region A inframe n+1, is multiplied in frame n+2 (FIG. 5(i)) by the digital gainset based on the luminance detection result of the region B.

In and after frame n+3, the data obtained by the exposure with theexposure condition set based on the luminance detection result of theregion B, and multiplication by the digital gain set based on theluminance detection result of the region B is output.

For this reason, the extracted region is switched between frame n+2 andframe n+3, and the region B is extracted in and after frame n+3.

Accordingly, there is a delay of four frame periods, from the beginningof the frame in which the extracted region switching is instructed (FIG.5(k)), to the extracted region switching.

In contrast, according to the present invention, the delay from thebeginning of the frame in which the extracted region switching isinstructed (FIG. 4(k)), to the extracted region switching (FIG. 4(a)) isthree frame periods.

Also, in the case of FIGS. 5(a) to (k), the data output in frame n+2 isobtained by adjustment based on the exposure condition set based on theluminance detection result of the region A, and the digital gain setbased on the luminance detection result of the region B, so that it isnot ensured that the brightness is properly controlled, and thebrightness may differ from the preceding or following frame.

In contrast, in the case of FIGS. 4(a) to (k), before the extractedregion switching, an image obtained as a result of the exposure with theexposure condition set based on the luminance detection result of theregion A, and the multiplication by the digital gain set based on theluminance detection result of the same region A is output; after theextracted region switching, an image obtained as a result of theexposure with the exposure condition set based on the luminancedetection result of the region B, and the multiplication by the digitalgain set based on the luminance detection result of the same region B isoutput. That is, both before and after the region switching, thebrightness of the image is properly adjusted.

In the example shown in FIGS. 4(a) to (k), the digital gain is set inthe digital gain application unit 40 in the same frame as that in whichthe computation of the set values is made, and the use of the thus-setdigital gain is deferred by one frame in the digital gain applicationunit 40; but as shown in FIGS. 6(a) to (k), the digital gain may be setin the digital gain application unit 40 in the frame immediatelyfollowing the frame in which the computation of the set values is made,and the digital gain application unit 40 may immediately use the newlyset digital gain for the multiplication. That is, the delaying processmay be performed in the control unit 100, instead of the digital gainapplication unit 40.

In the embodiment described above, the luminance detection unit detectsthe luminance of all the regions in each frame; but the luminancedetection unit may detect the luminance of only those regions which arescheduled to be extracted or which have a possibility of beingextracted. For example, in the example described above, in the casewhere the regions R1 to R6 are successively selected, and the image istaken out from the selected region, only the regions R1 to R6 may betreated as the region which is currently extracted or the regions whichhave a possibility of being extracted, and the luminance detection ineach frame may be made only for these regions.

In the embodiment described above, the regions (R1 to R6, or the like)which are to be extracted are shown to be non-overlapping with eachother; but images of the regions which overlap with each other at theedges may be generated, and, at the time of combination, weightedaddition or selection may be made for the overlapping parts.

In the embodiment described above, the images of different regionssuccessively extracted by the extraction unit are combined by the imagecombining unit; but the present invention is not limited to this scheme:for example, the present invention is applicable to a configuration inwhich the image of the extracted region is displayed as is (withoutbeing connected with the images of other regions). In this case, if aregion to be extracted next is known when a certain region is beingextracted, the luminance detection may be made for that region known tobe extracted next. Also, if it is known, when a certain region is beingextracted, that the region to be extracted next is one of a plurality ofknown regions (e.g., one of the regions neighboring the region currentlybeing extracted), the luminance detection may be made for theabove-mentioned plurality of known regions.

Also, where a plurality of regions are connected to form a singlecombined image, the regions forming parts of the combined image may besuccessively replaced so that the range occupied by the combined imageis moved gradually. For instance, from the state in which the combinedimage is formed of the regions R1 to R6 as in FIG. 2(a), the regions R1and R2 at the left end may be removed, and the regions R7 and R8neighboring on the right side may be added in place, so as to form acombined image consisting of the regions R3 to R8; and the similarprocesses may be repeated thereafter.

In this case too, when the direction of the movement is known, theluminance detection in each frame may be made, not only of the regionsR1 to R6 which are currently combined, but also of the regions R7, andR8, which are scheduled to be combined next. If the direction of themovement is not known, but it is known that adjacent regions on eitherside are successively added, and the regions on the opposite side areremoved, the luminance detection in each frame may be made not only ofthe regions which are currently combined, but also of the adjacentregions (the regions which may be combined).

In the embodiment described above, the switching is made between regionsoccupying different positions in the image; but the present inventioncan be applied to cases where switching is made between one region, andanother region which includes, as its part, the above-mentioned oneregion (between a narrower region and a wider region). A zoom processcan be achieved by selecting and displaying the images of narrower andwider regions in succession.

The present invention is applicable to cases in which the switching ismade between a plurality of regions which are at the same position(central position) and have different inclination (tilt) angles. Displaywith different tilt angles can be made by selecting and displaying theregions with different tilt angles in succession.

As has been described, according to the present invention, it ispossible to perform exposure control that is stable and causes littlevariation in the luminance of the image, even when the switching of theregion extracted from the image is performed, and the delay from theinstruction for the extracted region switching, to the execution of theswitching is short.

Second Embodiment

In the first embodiment described above, the regions extracted from theimage and the regions forming units in the luminance detection areidentical; but the present invention is applicable to cases where theregions extracted from the image and the regions forming units of theluminance calculation are different. In the following description, forthe purpose of distinction from the extracted region, the region forminga unit of the luminance calculation is referred to as a “section”. Whenthe sections, that is, the regions forming units of the luminancecalculation, are smaller than extracted regions of the image, and eachextracted region extends over a plurality of sections, that is, when theplurality of sections is at least partly included in the extractedregion, the luminance of the extracted region may be calculated based onthe luminance of the plurality of sections. For example, the capturedimage may be divided into a plurality of rectangular sections (blocks),and the luminance of each block may be detected, and the luminance ofeach of the blocks may be used to calculate the luminance of theextracted region in which the above-mentioned blocks are at least partlyincluded.

FIG. 7(a) shows an example in which the image is divided by grid lines,into a plurality of rectangular sections (blocks) BK, and an extractedregion ER is set to extend over a plurality of the rectangular blocks.In FIGS. 7(b) to (d), the blocks BKe which are used for the calculationof the luminance of the extracted region ER are indicated by hatching.In the example of FIG. 7(b), only those blocks BKe which are entirelyincluded in the extracted region ER are used for the calculation of theluminance of the extracted region ER; in the example of FIG. 7(c), onlythose blocks BKe which each have at least a half thereof included in theextracted region ER are used for the calculation of the luminance of theextracted region ER; in the example of FIG. 7(d), all of the blocks BKewhich are at least partly included in the extracted region ER are usedfor the calculation of the luminance of the extracted region ER.

The calculation of the luminance of the extracted region based on theluminance of the blocks may be simple averaging, or weighted averaging.In the case of the weighted averaging, the weights may be assigned sothat they correspond to the proportion of each block which is includedin the extracted region.

As has been described, according to the present embodiment, theluminance of the extracted region is calculated by using the luminanceof each of the rectangular sections (blocks) formed by dividing thecaptured image, so that the luminance of the extracted region can becalculated flexibly, regardless of the shape of the extracted region,and even if the shape of the extracted region changes with time.

So far, the image processing apparatus of the present invention has beendescribed; but the image processing method implemented in theabove-described image processing apparatus also constitutes part of thepresent invention. Also part or the entirety of the constituent elementsof the above-described image processing apparatus, or part or theentirety of the processes in the above-described image processing methodmay be realized by software, i.e., a programmed computer. Accordingly, aprogram for causing a computer to execute part or the entirety of theconstituent elements of the above-described image processing apparatus,or part or the entirety of the processes of the above-described imageprocessing method, and a computer-readable recording medium in which theprogram is recorded also constitute part of the present invention.

REFERENCE CHARACTERS

12 wide-angle image acquisition unit, 14 image processing apparatus, 20wide-angle lens, 30 image capture unit, 40 digital gain applicationunit, 50 luminance detection unit, 60 image generating unit, extractionunit, 80 image combining unit, 100 control unit.

1. An image processing apparatus comprising: a digital gain applicationunit for multiplying, by a digital gain, a captured image signal outputfrom an image capture unit which captures images in units of frameperiods, to generate a luminance-adjusted captured image signal; aluminance detection unit for detecting luminance of each of a pluralityof regions which respectively form parts of a captured image representedby the captured image signal generated by said digital gain applicationunit; an extraction unit for selecting one of the plurality of regionsof the captured image, in accordance with designation informationdesignating a region to be extracted, extracting an image of theselected region, and performing distortion correction; and a controlunit for, on a basis of the luminance detected by said luminancedetection unit, setting a condition of exposure in said image captureunit, and setting the digital gain used in said digital gain applicationunit; wherein when the designation information is changed frominformation designating a first region to information designating asecond region, in a first frame period, said control unit changes theluminance used for setting said condition of the exposure and saiddigital gain in a frame period immediately following said first frameperiod, from the luminance of said first region to the luminance of saidsecond region, and instructs said extraction unit to change the regionextracted from the captured image signal, from said first region to saidsecond region, upon expiration of three frame periods after said firstframe period.
 2. The image processing apparatus as set forth in claim 1,wherein said digital gain application unit multiplies the captured imagesignal generated in said image capture unit, as a result of the exposurein each frame period, by the gain in a next frame period, to output theluminance-adjusted captured image signal; said control unit, based onthe luminance detected by said luminance detection unit in said eachframe period, sets said condition of the exposure at said image captureunit in a frame period two frame periods after said each frame period,and the digital gain used for the multiplication at said digital gainapplication unit in a frame period three frame periods after said eachframe period; said control unit starts determination of said conditionof the exposure and said digital gain based on the luminance of saidsecond region in said first frame period, and causes said extractionunit to extract the image of said second region in and after the frameperiod three frames after said first frame period.
 3. The imageprocessing apparatus as set forth in claim 1, wherein said luminancedetection unit detects the luminance of each of all the regionsconsisting of the region which is currently extracted in each frameperiod, and regions which have a possibility of being extracted, amongsaid plurality of regions.
 4. The image processing apparatus as setforth in claim 1, wherein said luminance detection unit detects theluminance of each of a plurality of sections formed by dividing thecaptured image, and calculates the luminance of each of said pluralityof regions based on the luminance of one or more of the sections whichare at least partly included in said each of the regions.
 5. The imageprocessing apparatus as set forth in claim 2, wherein said control unitperforms computation for determination of said condition of the exposureand determination of said digital gain in a frame period immediatelyfollowing said each frame period, and performs computation fordetermination of said condition of the exposure and determination ofsaid digital gain based on a result of detection of the luminance ofsaid second region, in and after the frame period immediately followingsaid first frame period.
 6. The image processing apparatus as set forthin claim 5, wherein said control unit sets said condition of theexposure and said digital gain having been determined, in said imagecapture unit and said digital gain application unit, in the frame periodimmediately following said each frame period; and said digital gainapplication unit delays use of said digital gain having been set, by oneframe period.
 7. The image processing apparatus as set forth in claim 5,wherein said control unit sets said condition of the exposure havingbeen determined, in said image capture unit in the frame periodimmediately following said each frame period, and sets said digital gainhaving been determined in said digital gain application unit, in theframe period two frame periods after said each frame period.
 8. Theimage processing apparatus as set forth in claim 1, wherein the settingof said condition of the exposure and the setting of said digital gainare performed in a vertical blanking period in each frame period.
 9. Theimage processing apparatus as set forth in claim 1, further comprisingan image generating unit for receiving the captured image signal outputfrom said digital gain application unit, and performing image processingincluding color interpolation, color correction, and edge enhancement;wherein said extraction unit performs extraction from an image havingundergone the image processing in said image generating unit.
 10. Theimage processing apparatus as set forth in claim 1, further comprisingan image combining unit for combining the images of the plurality ofregions output from said extraction unit, to generate a singlecontinuous image.
 11. An image processing method comprising: a digitalgain application step for multiplying, by a digital gain, a capturedimage signal output from an image capture unit which captures images inunits of frame periods, to generate a luminance-adjusted captured imagesignal; a luminance detection step for detecting luminance of each of aplurality of regions which respectively form parts of a captured imagerepresented by the captured image signal generated by said digital gainapplication step; an extraction step for selecting one of the pluralityof regions of the captured image, in accordance with designationinformation designating a region to be extracted, extracting an image ofthe selected region, and performing distortion correction; and a controlstep for, on a basis of the luminance detected by said luminancedetection step, setting a condition of exposure in said image captureunit, and setting the digital gain used in said digital gain applicationstep; wherein when the designation information is changed frominformation designating a first region to information designating asecond region, in a first frame period, said control step changes theluminance used for setting said condition of the exposure and saiddigital gain in a frame period immediately following said first frameperiod, from the luminance of said first region to the luminance of saidsecond region, and instructs said extraction step to change the regionextracted from the captured image signal, from said first region to saidsecond region, upon expiration of three frame periods after said firstframe period.
 12. (canceled)
 13. A computer-readable recording medium inwhich a program for causing a computer to execute processes of the stepsin claim 11 is recorded.
 14. The image processing apparatus as set forthin claim 2, wherein said luminance detection unit detects the luminanceof each of all the regions consisting of the region which is currentlyextracted in each frame period, and regions which have a possibility ofbeing extracted, among said plurality of regions.
 15. The imageprocessing apparatus as set forth in claim 2, wherein said luminancedetection unit detects the luminance of each of a plurality of sectionsformed by dividing the captured image, and calculates the luminance ofeach of said plurality of regions based on the luminance of one or moreof the sections which are at least partly included in said each of theregions.
 16. The image processing apparatus as set forth in claim 2,wherein the setting of said condition of the exposure and the setting ofsaid digital gain are performed in a vertical blanking period in eachframe period.
 17. The image processing apparatus as set forth in claim2, further comprising an image generating unit for receiving thecaptured image signal output from said digital gain application unit,and performing image processing including color interpolation, colorcorrection, and edge enhancement; wherein said extraction unit performsextraction from an image having undergone the image processing in saidimage generating unit.
 18. The image processing apparatus as set forthin claim 2, further comprising an image combining unit for combining theimages of the plurality of regions output from said extraction unit, togenerate a single continuous image.